Method and system for providing haptic effects based on information complementary to multimedia content

ABSTRACT

The present disclosure is generally directed to systems and methods for providing haptic effects based on information complementary to multimedia content. For example, one disclosed method includes the steps of receiving multimedia data comprising multimedia content and complementary data, wherein the complementary data describes the multimedia content, determining a haptic effect based at least in part on the complementary data, and outputting the haptic effect while playing the multimedia content.

REFERENCE TO RELATED APPLICATION

This application is a continuation application of and claims priority toco-pending U.S. application Ser. No. 15/479,398, filed Apr. 5, 2017,entitled “Method and System for Providing Haptic Effects Based onInformation Complementary to Multimedia Content,” which is acontinuation application of and claims priority to U.S. application Ser.No. 14/145,650, filed Dec. 31, 2013, entitled “Method and System forProviding Haptic Effects Based on Information Complementary toMultimedia Content,” which claims priority to U.S. Provisional PatentApplication No. 61/874,933, filed Sep. 6, 2013, entitled “Audio toHaptics,” the entirety of each of which is hereby incorporated herein byreference.

This application is related to U.S. patent application Ser. No.14/078,438, entitled “Systems and Methods for Generating Haptic EffectsAssociated with Transitions in Audio Signals,” filed Nov. 12, 2013, theentirety of which is hereby incorporated herein by reference.

This application is related to U.S. patent application Ser. No.14/078,442, entitled “Systems and Methods for Generating Haptic EffectsAssociated with an Envelope in Audio Signals,” filed Nov. 12, 2013, theentirety of which is hereby incorporated herein by reference.

This patent application is related to U.S. patent application Ser. No.14/078,445, entitled “Systems and Methods for Generating Haptic EffectsAssociated with Audio Signals,” filed Nov. 12, 2013, the entirety ofwhich is hereby incorporated herein by reference.

FIELD

The present disclosure generally relates to methods and systems forproviding haptic effects based on multimedia data, and more particularlyto methods and systems for providing haptic effects based on informationcomplementary to multimedia content.

BACKGROUND

Advances in multimedia technology over recent years have continuouslyserved to enhance and provide a more immersive viewing experience forusers. Today's televisions have larger screens, high-definition screenresolutions, and 3D capabilities. In addition, handheld devices nowregularly have high resolution screens with vibrant colors and access tonumerous sources of content through the Internet. Similarly,advancements in audio technologies such as improved surround soundsystems, higher fidelity home theater speakers, and high definitionaudio further enhance users' multimedia viewing experience. In additionto advancements in video and audio technologies, haptic effects are usedto further immerse users in the viewing experience through the viewers'sense of touch. Haptically-enabled video game console controllers thatprovide haptic effects corresponding to events in a video game arepervasive examples of the use of haptics to enhance user experience.Similarly, most handheld mobile devices (e.g., cell phones, tabletcomputers) today contain hardware that permits applications to providehaptic effects.

Employing haptics to enhance the playback of audio and/or video contentis also well known. In some cases, haptic designers compile haptictracks—a specific set of haptic effects to be output at particular timesduring the playback of the multimedia content—that are used by systemsor devices playing the multimedia file to provide haptic effects. Whilean advantage of this method is that it yields haptic tracks specificallytailored to the multimedia content, it requires employing a hapticdesigner to manually design haptic tracks for each multimedia contentitem. Alternatives to manual design include automated processes that usealgorithms to analyze the audio and/or video content to determine hapticeffects to be output. Examples of such techniques include Acoustic EventDetection (“AED”), described in above-referenced U.S. patent applicationSer. No. 14/078,445, Blind Source Separation (“BSS”), also described inabove-referenced U.S. patent application Ser. No. 14/078,445, and imagepattern matching. However, these known automated haptificationalgorithms suffer from inaccuracy and heavy processing requirements. Inaddition, the automated haptification algorithms also have a tendency tobe over-inclusive—haptifying events that are not desirable and oftenresult in an overly noisy haptic track (e.g., typing on a keyboard,shouting, music)—and under-inclusive—failing to haptify key events inthe multimedia content.

SUMMARY

The present disclosure is generally directed to a method comprisingreceiving multimedia data comprising multimedia content andcomplementary data, wherein the complementary data describes themultimedia content, determining a haptic effect based at least in parton the complementary data, and outputting the haptic effect whileplaying the multimedia content. In another embodiment, a computerreadable medium comprises program code for causing a processor toperform such a method.

These illustrative embodiments are mentioned not to limit or define theinvention, but to provide examples to aid understanding thereof.Illustrative embodiments are discussed in the Detailed Description andfurther description of the invention is provided therein. Advantagesoffered by various embodiments of this invention may be furtherunderstood by examining this specification.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features, aspects, and advantages according to thepresent disclosure are better understood when the following DetailedDescription is read with reference to the accompanying figures, wherein:

FIGS. 1A and 1B are block system diagrams according to embodiments ofthe present disclosure.

FIGS. 2A-H are block diagrams of multimedia data according toembodiments of the present disclosure.

FIG. 3 is a flow diagram according to embodiments of the presentdisclosure.

FIG. 4 is a flow diagram according to embodiments of the presentdisclosure.

DETAILED DESCRIPTION

Embodiments according to this disclosure provide methods and systems forproviding haptic effects based on multimedia data, and more particularlyto methods and systems for providing haptic effects based on informationcomplementary to multimedia content (referred to hereinafter as“complementary information”). As used herein, “multimedia data”comprises both multimedia content and complementary information.

Illustrative Embodiment

In one illustrative embodiment, a family has a home theater systemcomprising a television, an Internet-capable and a haptic-enabled A/Vreceiver, speakers, and a haptic couch. The family regularly watchesmovies using an Internet-based movie streaming service through which theA/V receiver receives a multimedia data stream comprising a selectedmovie as well as complementary information (e.g. metadata tags, closedcaption information) for that movie. In one particular instance, thefamily chooses to watch a World War I action/drama film. As the datastream for the movie begins to arrive, the A/V receiver determines thata metadata container, commonly known as a tag, is present. Analyzing thetag, the A/V receiver determines the presence of the text “war” in agenre field of the tag and, based on the genre, determines that a haptictheme increasing the intensity of the output of haptic effects above adefault level will be applied. In addition, the A/V receiver providesthe genre to an automated haptification algorithm that analyzes theaudio and video tracks of the movie to identify patterns for which togenerate haptic effects. In the present example, the A/V receiverfocuses on patterns typically found in war films, such as explosions,gunfire, and the like and generates haptic effects based on thealgorithm. In addition to analyzing the audio and video tracks, the A/Vreceiver also analyzes closed caption text for keywords. For example, ifthe A/V receiver finds the word “explosion” at a particular timestamp inthe closed caption data, the A/V receiver inputs this information intothe automated algorithm. The algorithm then looks for an explosionpattern around that timestamp and generates a corresponding hapticeffect, such as an intense vibration, to correspond with the explosion.As the movie plays, the A/V receiver sends signals to the haptic sofacausing it to output the haptic effects at times corresponding to eventsoccurring within the film.

This illustrative embodiment is merely an example, and multiple otherembodiments of the present invention may be implemented as describedherein.

Illustrative Device

Referring now to the drawings in which like numerals indicate likeelements throughout the several figures, FIGS. 1A and 1B are blockdiagrams of systems for providing haptic effects based on complementaryinformation according to embodiments of the disclosure.

FIG. 1A depicts a system 100 comprising the various components. In someembodiments, the system 100 shown in FIG. 1A is a home entertainmentsystem comprising a television, an A/V receiver, a plurality ofspeakers, a gaming console with gaming controllers, a universal remotecontrol, haptically-enabled furniture, and/or wearable haptic outputdevices. In one particular embodiment, depicted by FIG. 1B, system 100comprises a television (display 116), an A/V receiver 122, speakers 120,a remote control 114, and a haptically-enabled couch 124. In otherembodiments, system 100 comprises a handheld device, such as a mobilephone, a personal digital assistant (PDA), tablet computer, laptopcomputer, palmtop computer, or a handheld navigation system. In stillother embodiments, system 100 may comprise a kiosk, an automotive headunit, or similar systems.

Embodiments of the present disclosure can be implemented in combinationwith, or may comprise combinations of: digital electronic circuitry,computer hardware, firmware, and software. The system 100 shown in FIGS.1A and 1B comprises a processor 110. The processor 110 receives inputsignals and generates signals for communication, display, and providinghaptic feedback. The processor 110 includes or is in communication withone or more computer-readable media, such as memory 112, which maycomprise random access memory (RAM).

The processor 110 executes computer-executable program instructionsstored in memory 112, such as executing one or more computer programsfor messaging or for generating haptic feedback. Processor 110 maycomprise a microprocessor, a digital signal processor (DSP), anapplication-specific integrated circuit (ASIC), one or more fieldprogrammable gate arrays (FPGAs), or state machines. The processor mayfurther comprise a programmable electronic device such as a PLC, aprogrammable interrupt controller (PIC), a programmable logic device(PLD), a programmable read-only memory (PROM), an electronicallyprogrammable read-only memory (EPROM or EEPROM), or other similardevices.

Memory 112 comprises a computer-readable media that may storeinstructions, which, when executed by the processor 110, cause it toperform various steps, such as those described herein. Embodiments ofcomputer-readable media may comprise, but are not limited to, anelectronic, optical, magnetic, or other storage or transmission devicecapable of providing the processor 110 with computer-readableinstructions. Other examples of media comprise, but are not limited to,a floppy disk, CD-ROM, magnetic disk, memory chip, ROM, RAM, ASIC,configured processor, all optical media, all magnetic tape or othermagnetic media, or any other medium from which a computer processor canread. Also, various other devices may include computer-readable media,such as a router, private or public network, or other transmissiondevice. The processor 110, and the processing, described may be in oneor more structures, and may be dispersed through one or more structures.

The system 100 is able to communicate with and/or receive communicationsfrom a network using a network interface 122. In some embodiments,network interface 122 is an Ethernet card. In other embodiments, networkinterface 122 is a module for receiving and decoding cable TV and/orsatellite TV signals housed within an A/V receiver, a cable TV set-topbox, a satellite TV set-top box, a television, or another component of asystem 100. However, in embodiments of the present invention, networkinterface 122 may be any component that allows communication with anetwork. For example, network interface 122 may be a wireless networkingdevice, or a module and antenna for communication with a cellularnetwork or a Wi-Fi network.

Referring still to FIGS. 1A and 1B, the system 100 also comprises a userinput device 114 in communication with the processor 110. For example,in some embodiments the user input device 114 may comprise atouchscreen. In such an embodiment, user input device 114 may sense userinteraction as well as the location of the interaction. One suchembodiment comprises a capacitance-based touchscreen. In otherembodiments, user input device 114 may comprise a button, switch,slider, or trackball. In still other embodiments, the system 100 maycomprise both a touch screen and an additional user input device 114.

In some embodiments, the user input device 114 may comprise a remotecontrol, gaming controller, or similar device in communication withprocessor 110 by way of a wired or a wireless connection (e.g.,infrared, Bluetooth, Zigbee, or any other wireless communicationstechnology). On such embodiment comprises a remote control for anaudio/video receiver that communicates with the audio/video receiverusing Bluetooth.

The system 100 also comprises a display 116. Display 116 is incommunication with processor 110 and is configured to display outputfrom the processor 110 to the user. In one embodiment, the display 116comprises a television. In another embodiment, the device 102 comprisesa liquid crystal display (LCD) disposed beneath the user input device114. In some embodiments, the display 116 and user input device 114 maycomprise a single, integrated component, such as a touch-screen LCD. Insome embodiments, device 102 may not comprise a display.

In the embodiment shown in FIGS. 1A, 1B and 2, haptic output devices 118and are in communication with the processor 110 and are configured toprovide one or more haptic effects. For example, in one embodiment, whenan actuation signal is provided to haptic output device 118 by theprocessor 110, the respective haptic output device(s) 118, output ahaptic effect based on the actuation signal. For example, in theembodiment shown, the processor 110 is configured to transmit a hapticoutput signal to haptic output device 118 comprising an analog drivesignal. In some embodiments, the processor 110 is configured to transmita command to haptic output device 118, wherein the command includesparameters to be used to generate an appropriate drive signal to causethe haptic output device 118 to output the haptic effect. In otherembodiments, different signals and different signal types may be sent toeach of one or more haptic output devices. For example, in someembodiments, a processor may transmit low-level drive signals to drive ahaptic output device to output a haptic effect. Such a drive signal maybe amplified by an amplifier or may be converted from a digital to ananalog signal, or from an analog to a digital signal using suitableprocessors or circuitry to accommodate the particular haptic outputdevice being driven.

A haptic output device, such as haptic output device 118, can be anycomponent or collection of components that is capable of outputting oneor more haptic effects. For example, a haptic output device can be oneof various types including, but not limited to, an eccentric rotationalmass (ERM) actuator, a linear resonant actuator (LRA), a piezoelectricactuator, a voice coil actuator, an electro-active polymer (EAP)actuator, a memory shape alloy, a pager, a DC motor, an AC motor, amoving magnet actuator, an E-core actuator, a smartgel, an electrostaticactuator, an electrotactile actuator, a deformable surface, anelectrostatic friction (ESF) device, an ultrasonic friction (USF)device, or any other haptic output device or collection of componentsthat perform the functions of a haptic output device or that are capableof outputting a haptic effect. Multiple haptic output devices ordifferent-sized haptic output devices may be used to provide a range ofvibrational frequencies, which may be actuated individually orsimultaneously. Various embodiments may include a single or multiplehaptic output devices and may have the same type or a combination ofdifferent types of haptic output devices. In some embodiments, one ormore haptic output devices are directly or indirectly in communicationwith electronic device, such as via wired or wireless communication. Inone embodiment, the electronic device can be placed in a vehicle or isintegrated into a vehicle and one or more haptic output devices areembedded into the vehicle. For example, one or more haptic outputdevices may be embedded in a seat, steering wheel, pedal, etc. of thevehicle. In some embodiments, instead of having haptic output device 118or in addition to having haptic output device 118, the system 100 hasone or more other output devices. For example, system 100 may have aspeaker and/or a display. In one embodiment, the system 100 has one ormore haptic output devices, one or more speakers, and one or moredisplays. Numerous other embodiments are disclosed herein and variationsare within the scope of this disclosure.

In various embodiments, one or more haptic effects may be produced inany number of ways or in a combination of ways. For example, in oneembodiment, one or more vibrations may be used to produce a hapticeffect, such as by rotating an eccentric mass or by linearly oscillatinga mass. In some such embodiments, the haptic effect may be configured toimpart a vibration to the entire electronic device or to only onesurface or a limited part of the electronic device. In anotherembodiment, friction between two or more components or friction betweenat least one component and at least one contact may be used to produce ahaptic effect, such as by applying a brake to a moving component, suchas to provide resistance to movement of a component or to provide atorque. In order to generate vibration effects, many devices utilizesome type of actuator and/or other haptic output device. Known hapticoutput devices used for this purpose include an electromagnetic actuatorsuch as an Eccentric Rotating Mass (“ERM”) in which an eccentric mass ismoved by a motor, a Linear Resonant Actuator (“LRA”) in which a massattached to a spring is driven back and forth, or a “smart material”such as piezoelectric, electro-active polymers or shape memory alloys.

In other embodiments, deformation of one or more components can be usedto produce a haptic effect. For example, one or more haptic effects maybe output to change the shape of a surface or a coefficient of frictionof a surface. In an embodiment, one or more haptic effects are producedby creating electrostatic forces and/or ultrasonic forces that are usedto change friction on a surface. In other embodiments, an array oftransparent deforming elements may be used to produce a haptic effect,such as one or more areas comprising a smartgel. Haptic output devicesalso broadly include non-mechanical or non-vibratory devices such asthose that use electrostatic friction (ESF), ultrasonic surface friction(USF), or those that induce acoustic radiation pressure with anultrasonic haptic transducer, or those that use a haptic substrate and aflexible or deformable surface, or those that provide projected hapticoutput such as a puff of air using an air jet, and so on. In someembodiments, a haptic effect is a kinesthetic effect. U.S. patentapplication Ser. No. 13/092,484 describes ways that one or more hapticeffects can be produced and describes various haptic output devices. Theentirety of U.S. patent application Ser. No. 13/092,484, filed Apr. 22,2011, is hereby incorporated by reference.

It will be recognized that any type of input synthesis method may beused to generate an interaction parameter for one or more haptic effectsignals including, but not limited to, the method of synthesis exampleslisted in TABLE 2 below. A drive signal may be applied to a hapticactuator according to the interaction parameter. Numerous otherembodiments are disclosed herein and variations are within the scope ofthis disclosure.

TABLE 2 METHODS OF SYNTHESIS Additive synthesis - combining inputs,typically of varying amplitudes Subtractive synthesis - filtering ofcomplex signals or multiple signal inputs Frequency modulationsynthesis - modulating a carrier wave signal with one or more operatorsSampling - using recorded inputs as input sources subject tomodification Composite synthesis - using artificial and sampled inputsto establish a resultant “new” input Phase distortion - altering thespeed of waveforms stored in wavetables during playback Waveshaping -intentional distortion of a signal to produce a modified resultResynthesis - modification of digitally sampled inputs before playbackGranular synthesis - combining of several small input segments into anew input Linear predictive coding - similar technique as used forspeech synthesis Direct digital synthesis - computer modification ofgenerated waveforms Wave sequencing - linear combinations of severalsmall segments to create a new input Vector synthesis - technique forfading between any number of different input sources Physical modeling -mathematical equations of the physical characteristics of virtual motion

In further embodiments, one or more haptic output devices 118 may beused to implement a wearable haptic device (e.g. a wristband, belt, orshirt). In other embodiments, one or more haptic output devices 118 maybe used to implement a haptically-enabled furniture such as a chair or acouch.

As mentioned above, in one embodiment system 100 of FIG. 1A may be ahome entertainment system. For example, in one embodiment depicted byFIG. 1B, system 100 comprises a television (display 116), an A/Vreceiver 122, speakers 120, a remote control 114, and ahaptically-enabled couch 124. The A/V receiver 122 comprises processor110, memory 112, and memory 122. User input device 114 comprises aremote control in communication with the processor 110 of A/V receiver122. A/V receiver operates to play multimedia content residing in memory112 or multimedia content received through network interface 122. In oneembodiment, playing multimedia content comprise processor 110transmitting video signals to display 116 causing video to be displayedand audio signals to speakers 120 causing audio to be played as is wellknown in the art. In another embodiment, playing multimedia contentfurther comprises transmitting haptic signals to haptic output devices118 of haptically-enabled couch 124 to cause haptic effects to be outputto viewers sitting thereon.

In further embodiments, system 100 may comprise sensors in communicationwith processor 110. In one embodiment, system 100 may be configured touse one or more sensors to detect viewer information. For example,system 100 may be configured to detect viewer location, position,movement, and/or viewer biometric information. For example, remotecontrol 114 may comprise a proximity sensor configured to detect thedistance of the remote control from A/V receiver 122. In one suchembodiment, the remote control 114 is configured to communicate detecteddistance information to processor 110 of A/V receiver 122. In anotherembodiment, system 100 may further comprise a camera and software fordetermining viewer information based on video captured by the videocamera. In another embodiment, wearable devices, (e.g. a wearable hapticdevice) comprise one or more proximity sensors configured to detect thedistance of the viewer wearing the device from A/V receiver 122 and tocommunicate detected distance information to processor 110 of A/Vreceiver 122. In some embodiments, wearable devices of system 100 maycomprise sensors configured to detect motion (e.g. an accelerometer) andto communicate detected motion to processor 110 of A/V receiver 122.Based on this information, processor 110 may determine the movement of auser of system 100 according to methods known by one of ordinary skillin the art. In another embodiment, the wearable devices of system 100comprise sensors configured to detect biometric data of the wearer suchas heart rate or body temperature. System 100 may then analyze thebiometric data to determine the stress level of a viewer and/orinvoluntary bodily responses of the viewer to multimedia content beingplayed.

In another embodiment, a piece of furniture (e.g. haptically-enabledcouch 124) comprises sensors to detect the presence and movements of oneor more users of system 100. For example, pressure sensors may belocated at various positions on haptically-enabled couch 124 andconfigured to communicate detected pressure to processor 110 of A/VReceiver 122 permitting the system 100 to detect whether and how a useris sitting on the couch 124 as well as movements of the user based onchanges in the pressure detected by the pressure sensors according tomethods known by one of ordinary skill in the art. Furthermore, thepresent disclosure contemplates all sensors and methods known to thoseof ordinary skill in the art to detect position and/or movement of auser of system 100.

Complementary Information

Complementary information is now discussed with reference to FIGS. 2A-H.For the purposes of the present disclosure, complementary information isinformation corresponding to multimedia content (e.g., any audio and/orvideo signal, file, or other form of audio and/or video data). In someembodiments, the complementary information generally describes themultimedia content, such as subtitle data, closed caption data (whichmay include subtitle data), a script, or a metadata tag. In anotherembodiment, complementary information may be annotations or commentsfrom viewers (e.g. comments posted by members of an on-line videowebsite) about the video as a whole or applicable to particular times ofthe multimedia content. Complementary information may take many forms indifferent embodiments, and may include other types of information, suchas title or chapter information. Complementary information is oftenincluded (embedded) with the multimedia content it describes in a singlemultimedia file, which in various embodiments may also be referred to asan A/V file, a video file, or an audio file.

FIG. 2A is a block diagram of multimedia data according to oneembodiment of the present disclosure. In this embodiment, multimediadata 200 comprises a multimedia file 201. In the present embodiment,multimedia file 201 is a video file comprising multimedia content 202(video) and complementary information in the form of subtitle/closedcaption data 204 corresponding to the multimedia content 202. In oneembodiment, subtitle/closed caption data 204 comprises text describingscenes and/or events visually depicted by multimedia content 202, textdescribing sounds heard in the audio of multimedia content 202, and/ortextual quotes of audible speech in the audio of multimedia content 202.

FIG. 2B is a block diagram of multimedia data according to oneembodiment of the present disclosure. In this embodiment, multimediadata 200 comprises a multimedia file 201. In the present embodiment,multimedia file 201 is a video file comprising multimedia content 202(video) and complementary information in the form of a tag 208 and ascript 206 corresponding to the multimedia content 202. Tags formultimedia content may comprise title, artist/author, genre, lengthdata, rating, and/or any other data describing the nature, content ororigin of the multimedia content (e.g. reviews by critics or otherviewers, list of songs used in a movie). One known example are ID3tags—metadata containers often used with MP3 files. In one embodiment,tag data is defined by an author of the multimedia content. In anotherembodiment, tag data is defined by a distributor of the multimediacontent. In still another embodiment, the tag data is defined by apurchaser and/or recipient of the multimedia content. In some suchembodiments, the tag is generated by the author, distributor, and/orrecipient.

FIG. 2C is a block diagram of multimedia data according to oneembodiment of the present disclosure. In this embodiment, multimediadata 200 comprises a multimedia file 201. In the present embodiment,multimedia file 201 is an audio file comprising multimedia content 202(audio) and complementary information in the form of tag 208corresponding to the multimedia content 202.

FIG. 2D is a block diagram of multimedia data according to oneembodiment of the present disclosure. In this embodiment, multimediadata 200 comprises a multimedia file 201 and a complementary informationfile 210. In the present embodiment, multimedia file 201 is a video filecomprising multimedia content 202 (video) and complementary informationin the form of a tag 208 corresponding to the multimedia content 202. Inthe present embodiment, complementary information file 210 comprisescomplementary information in the form of subtitle/closed caption data204 corresponding to multimedia content 202. In one embodiment,complementary information file 210 resides in the same memory asmultimedia file 201 (e.g., memory 112). In another embodiment, thecomplementary information file 202 may reside on a remote serveraccessible by system 100 over a network, such as the Internet. In oneembodiment, the complementary information file 210 corresponding to amultimedia file 201 is retrieved from a remote server based oninformation contained in tag 208 (e.g., movie name, song name,identification code, or any other identifying information).

FIG. 2E is a block diagram of multimedia data according to oneembodiment of the present disclosure. In this embodiment, multimediadata 200 comprises a multimedia file 201 and a complementary informationfile 210. In the present embodiment, multimedia file 201 is a video filecomprising multimedia content 202 (video) and complementary informationin the form of subtitle/closed caption data 204 corresponding tomultimedia content 202. In the present embodiment, complementaryinformation file 210 comprises complementary information in the form oftag data 208 corresponding to multimedia content 202. In one embodiment,complementary information file 210 resides in the same memory asmultimedia file 201 (e.g., memory 112). In another embodiment, thecomplementary information file 202 may reside on a remote serveraccessible by system 100 over a network, such as the Internet. In oneembodiment, the complementary information file 210 corresponding to amultimedia file 201 is retrieved from a remote server based oninformation in the multimedia file 201 (e.g. metadata, file name).

FIG. 2F is a block diagram of multimedia data according to oneembodiment of the present disclosure. In this embodiment, multimediadata 200 comprises a multimedia file 201. In the present embodiment,multimedia file 201 is a video file comprising multimedia content 202(video) and complementary information in the form of chapter data 212corresponding to the multimedia content 202. In one embodiment, chapterdata 212 comprises the titles of chapters of the multimedia content 202and the times at which each chapter begins.

FIG. 2G is a block diagram of multimedia data according to oneembodiment of the present disclosure. In this embodiment, multimediadata 200 comprises a multimedia file 201. In the present embodiment,multimedia file 201 is a video file comprising multimedia content 202(video). In the present embodiment, complementary information file 210comprises complementary information in the form of user comments 214corresponding to the multimedia content 202. In one embodiment,complementary information file 210 resides in the same memory asmultimedia file 201 (e.g., memory 112). In another embodiment, thecomplementary information file 202 may reside on a remote serveraccessible by system 100 over a network, such as the Internet. In oneembodiment, the complementary information file 210 corresponding to amultimedia file 201 is retrieved from a remote server based oninformation in the multimedia file 201 (e.g. metadata, file name). Inone embodiment, user comments 214 are comments from viewers (e.g.comments posted by members of an on-line video website) about the videoas a whole or applicable to particular times of the multimedia content202.

FIG. 2H is a block diagram of multimedia data according to oneembodiment of the present disclosure. In this embodiment, multimediadata 200 comprises a multimedia file 201. In the present embodiment,multimedia file 201 is a video file comprising multimedia content 202(video). In the present embodiment, complementary information file 210comprises complementary information in the form of user-provided haptickeywords/phrases 216 corresponding to the multimedia content 202. In oneembodiment, complementary information file 210 resides in the samememory as multimedia file 201 (e.g., memory 112). In another embodiment,the complementary information file 202 may reside on a remote serveraccessible by system 100 over a network, such as the Internet. In oneembodiment, the complementary information file 210 corresponding to amultimedia file 201 is retrieved from a remote server based oninformation in the multimedia file 201 (e.g. metadata, file name). Inone embodiment, user-provided haptic keywords/phrases 216 are providedby users of a website for improving or correcting the haptic tracksgenerated for the multimedia content 202. In one embodiment, theuser-provided haptic keywords/phrases 216 apply to the multimediacontent 202 as a whole. In another embodiment, the user-provided haptickeywords/phrases 216 apply to chapters or portions of the multimediacontent 202. In still another embodiment, the user-provided haptickeywords/phrases 216 apply to specific points of time during playback ofthe multimedia content 202.

Operation of an Illustrative System

FIG. 3 is a flow chart describing an exemplary embodiment for a methodfor providing haptic effects based on complementary information. FIG. 4,which is encompassed by step 304 of FIG. 3, is a flow chart describingan exemplary embodiment for a method for providing haptic effects basedon complementary information. In some embodiments, the steps in FIGS. 3and 4 may be implemented in program code that is executed by aprocessor, for example, the processor in a general purpose computer, amobile device, or server. In some embodiments, these steps may beimplemented by a group of processors, for example, a processor on amobile device and processors on one or more general purpose computers,such as servers. To aid in understanding how each of the steps may beperformed, the following description is provided in the context of theillustrative block diagrams of the system 100 shown in FIGS. 1A and 1B.However, embodiments according to the present disclosure may beimplemented in alternative embodiments.

Beginning at step 302, the processor 110 of system 100 receivesmultimedia data 200. In one embodiment, the multimedia data 200 is amultimedia file 201 residing in memory 112 and the multimedia data 200is received in response to an operating system operation (e.g.,open/read commands issued to a file system). In another embodiment, themultimedia data 200 may be streamed over a network connection throughnetwork interface 122 and buffered at system 110. Embodiments of thepresent disclosure contemplate all network connections (e.g. Wi-Fi,Ethernet, wired modem, wireless modem, etc.) and sources and methods ofstreaming content known to those of ordinary skill in the art, includingstreaming from any number of servers over the Internet and streamingfrom a server within a corporate intranet or home network. In anotherembodiment, the multimedia data 200 is downloaded over a network, suchas the Internet, and stored in memory 112.

In some embodiments, the multimedia content 202 and complementaryinformation 422 (collectively, “multimedia data”) are contained in asingle file or data stream. However, in other embodiments, themultimedia content 202 and the complementary information 422 areseparate files. As discussed above, in one such embodiment, themultimedia content 202 and the complementary information 422 reside indifferent locations. For example, in one embodiment, the multimediacontent 202 may reside in memory 112 of system 100 while thecorresponding complementary information 422 may reside at a serveraccessible on a network such as the Internet. At the time the multimediacontent 202 is accessed for analysis and/or playback, the system 100 mayretrieve the complementary information 422 from the server and load thecomplementary information into memory 112. In other embodiments, themultimedia content 202 and the complementary information 422 may resideat separate servers accessible through a network such as the Internet.In still further embodiments, the multimedia content 202 andcomplementary information 422 may reside as separate files on the sameserver. In one such embodiment, the system 100 receives the multimediacontent 202 from a server by way of streaming and downloads thecomplementary information 422 from the server on which it resides. Instill further embodiments, a multimedia file 201 comprises complementaryinformation 422 and a corresponding complementary information file 210comprises additional complementary information 422.

At step 304, haptic effects to be output during playback of themultimedia content 202 are determined based at least in part on thecomplementary information 422. According to the present disclosure,complementary information 422 may be used to determine haptic effects tobe output during playback of multimedia content 202 using three primarytechniques: 1) identifying haptic effects to be output based on keywordsand phrases; 2) providing contextual information to algorithms/processesfor automatically haptifying audio and/or video image; and 3)identifying a haptic theme based on the complementary information 422.In one embodiment, each of these techniques comprises analyzingcomplementary information 422 by comparing words and/or phrases thereinto one or more lists or databases of keywords and/or phrases mapped tohaptic effect identification information, context identificationinformation, and/or haptic theme information (referred to hereinafter as“keyword database(s)”). In one embodiment, the list or database maycontain information indicating whether particular keywords and/orphrases are applicable to all types of complementary information 422(e.g., subtitle/closed caption data 204, scripts 206, and/or tags 208),or a subset thereof. In another embodiment, separate lists or databasesof keywords or phrases applicable to particular types of complementaryinformation 422 are used. For example, in one embodiment, a firstdatabase of keywords and/or phrases is used with respect tosubtitle/closed caption data 204, and scripts 206, while a seconddatabase is used with respect to tags 208.

Identifying Predetermined Haptic Effects Based on Keywords/Phrases

In one embodiment, haptic effects are identified based onkeywords/phrases present in the subtitle/closed caption data 204 orscript 206. In one such embodiment, the subtitle/closed caption data 204is analyzed at element 412. In one embodiment, the subtitle/closedcaption data 204 and/or user-provided haptic keywords/phrases 216 areanalyzed by comparing words and/or phrases to a keyword database. Forexample, in one such embodiment comparing the subtitle/closed captiondata 204 and/or user-provided haptic keywords/phrases 216 to a keyworddatabase reveals that “explosion” is present in each, and that it ismapped to haptic effect identification information for an intensevibration haptic that dissipates over time until the effect finishes(emulating an explosion). In another exemplary embodiment, comparing thesubtitle/closed caption data 204 and/or user comments 214 to a keyworddatabase reveals that “gunshot” is present in each, and that it ismapped to haptic identification information for an intense jolt hapticeffect that is short in duration to mimic a gunshot. In anotherembodiment, the subtitle/closed caption data 204 comprises songtitles/types playing in the background of different scenes—that mayindicate the nature or genre of the scene—and comparing the songtitles/types to a keyword database reveals mappings for each songtitle/type (or a subset thereof) mapped to haptic identificationinformation for particular haptic effects. In some embodiments, thesehaptic effects mapping are selected in order to relay to the user thenature/genre of the scene. One of ordinary skill in the art wouldrecognize that the words or phrases that may be used to identify hapticeffects to be output during playback of a multimedia file 201 are notlimited to the examples provided by the present disclosure, but ratherextend to any number of words or phrases.

For each keyword or phrase found in a keyword database, the processor110 retrieves the corresponding haptic effect identification informationfrom the keyword database and associates it with a timestampcorresponding to the word or phrase in subtitle/closed caption data 204.In some embodiments, the processor 110 fully analyzes the complementaryinformation 422 prior to playback of the multimedia content 202. In onesuch embodiment the processor 110 compiles the haptic effectidentification information and corresponding timestamps into a haptictrack 414 that is embedded into a multimedia file 201 comprisingmultimedia content 202 in any manner known to one having ordinary skillin the art. In another embodiment, the haptic track 414 is stored as afile separate from multimedia file 201 in a memory (e.g. memory 112). Inother embodiments, especially where the processor 110 receives themultimedia content 202 through streaming, at element 412, the processor110 analyzes the complementary information 422 during playback of themultimedia content and temporarily stores the haptic effectidentification information (e.g., a partial haptic track 414) in amemory, such as memory 112. In one such embodiment, playback of themultimedia content 202 is delayed to sufficiently allow the processor toperform the analysis of the complementary information 422 and initiatethe output of haptic effects in a timely manner during playback of themultimedia content 202.

In some embodiments, the method of identifying haptic effects identifiedbased on keywords present in the subtitle/closed caption data 204 and/orscript data 206 is the sole technique employed. In such embodiments, theapplicable elements of FIG. 4 are multimedia data 200, complementaryinformation 422, elements 412 and 414, and haptic track 418. As will bediscussed in more detail below, in some embodiments two of thetechniques may be used in conjunction and, in still further embodiments,all three of the techniques may be used in conjunction.

Providing Context to Automated Haptification Algorithms

In some embodiments, complementary information 422 is used to providecontext to automated haptification algorithm(s), (such as one or moreaudio to haptics algorithms 406 and/or one or more video to hapticsalgorithms 408) for analyzing an audio track 402 and/or a video track404 of multimedia content 202 to identify haptic effects to be output.In one embodiment, events, type of scene, and or presence of certainelements in a scene are determined based on the presence of keywords incomplementary information 422. For example, in one embodiment, atelement 412, the processor 110 searches one or more keyword databasesfor words or phrases contained in the complementary information 422corresponding to a multimedia content 202. For each keyword or phrasefound in the keyword database, the processor 100 retrieves the contextidentification information, associates it with a timestamp correspondingto the word or phrase in complementary information 422, if available,and provides it to one or more automated haptification algorithms (406and or 408) by way of a data structure, function call, messaging, or anyother means known to one having ordinary skill in the art. For example,in one such embodiment the word “bomb” in complementary information 422is mapped to context identification information used for all types ofexplosions, thereby informing an automated haptification algorithm (406and/or 408) to look for an explosion. In another exemplary embodiment,the word “crime” in a genre field of a tag 208 or the word “violence” ina rating field of a tag 208 embedded in the multimedia content 202 mapsto context identification information used to instruct an automatedhaptification algorithm (406 and/or 408) to prioritize recognition ofgunshots, events related to an automobile chases (e.g., squealing tires,collisions), and/or other crime related events. In another exemplaryembodiment, the word “saxophone” in complementary information 422 ismapped to context identification information used to instruct anautomated haptification algorithm (406 and/or 408) to prioritizerecognition of a saxophone or similar instrument playing. In someembodiments, the detected keyword may be used to provide context to theautomated haptification algorithms (406 and/or 408) for the full lengthof the multimedia content 202. In other embodiments, the detected keymay be used to provide context to the automated haptification algorithms(406 and/or 408) for a subset of the multimedia content 202 (e.g. for aspecific scene or scenes of a movie). Educating the automated algorithmswith contextual information aids both processing efficiency and accuracyby prioritizing what patterns or events to look for in the audio and/orvideo. Accordingly, the timing and accuracy of the corresponding hapticeffects output by the algorithm based on its recognition of events isalso improved. One of ordinary skill in the art would recognize that thewords or phrases that may be used to provide context to automatedhaptification algorithms are not limited to the examples provided by thepresent disclosure, but rather extends to any words or phrases that mayprovide clues as to what events or elements are present in themultimedia content 202.

In some embodiments, the processor 110, at element 412, fully analyzesthe complementary information 422 to determine context identificationinformation and then executes automated haptification algorithms 406and/or 408 using the context identification information to process theaudio track 402 and/or video track 404 to generate haptic track 410prior to playback of the multimedia content 202. In one such embodiment,the processor 110 embeds the haptic track 410 generated by the automatedhaptification algorithm(s) a multimedia file 201 comprising themultimedia content 202 in any manner known to one having ordinary skillin the art. In another embodiment, the haptic track 410 is stored as afile separate from multimedia file 201 in a memory (e.g. memory 112). Inother embodiments, especially where the processor 110 receives themultimedia content 202 through streaming, the processor 110 analyzes thecomplementary information 422 and executes the automated haptificationalgorithm(s) (406 and/or 408), as described above, during playback ofthe multimedia content 202 and temporarily stores generated portions ofthe haptic track 410 in a memory, such as memory 112. In one suchembodiment, playback of the multimedia content 202 is delayed tosufficiently allow the processor 110 to perform the analysis of thecomplementary information 422 and execution of the automatedhaptification algorithm(s) (406 and/or 408), as described above, inorder to generate portions of the haptic track 410 in time to initiatethe output of haptic effects in a timely manner during playback of themultimedia content 202.

In some embodiments, the contextual information derived from thecomplementary information 422 is used to avoid haptifying or toattenuate the intensity of haptic effects corresponding to certainevents for which haptification is not desired. For example, in someembodiments the presence of “laughter” in subtitle/closed caption data204 is used to inform one or more automated haptification algorithms(406 and/or 408) to look for laughing or chuckling and avoid causing ahaptic effect to be output for that event. Similarly, in anotherembodiment the presence of “music” in the subtitle/closed caption data204 is used to inform one or more automated haptification algorithms(406 and/or 408) to look for playing of various instruments and/orsinging and to avoid causing the output of corresponding haptic effects.In another embodiment, a list of songs in a tag 208 is used to informautomated haptification algorithm 406 to look for and avoid haptifyingthe playback of the listed songs.

In some embodiments, the method of analyzing complementary information422 to provide context identification information to one or moreautomated haptification algorithm(s) (406 and/or 408), is the soletechnique employed. In such embodiments, the applicable elements of FIG.4 are multimedia data 200 (including audio track 402 and/or video track404), complementary information 422, one or more of automatedhaptification algorithms (406 and/or 408), element 412, and haptic track410.

In some embodiments, both the technique of identifying haptic effects tobe output based directly on keywords and the technique of providingcontextual information derived from complementary information 422 toautomatic haptification algorithms (406 and/or 408) for automaticgeneration of a haptic track are employed. In one such embodiment, theprocessor 110 combines automatically generated haptic track 410 andhaptic track 414 to produce combined haptic track 418. In one suchembodiment, processor 110 operates to insert directly derived hapticeffects from haptic track 414 into haptic track 410, and if necessaryremove conflicting auto-generated haptic effect from the automaticallygenerated haptic track 410, resulting in combined haptic track 418.

Haptic Themes

In some embodiments, complementary information 422 is used to determinea haptic theme to be applied to the haptic effects output duringplayback of a multimedia content 202 or to be applied to a certain sceneor portion thereof. For example, in one embodiment the processor 110, atelement 412, searches one or more keyword databases words or phrasescontained in the complementary information 422 for a multimedia content202. For each keyword or phrase found in the keyword database, theprocessor 110 retrieves the haptic theme information from the keyworddatabase and correlates it to the entirety or specific scenes of themultimedia content, thereby generating the one or more haptic themes 416to be applied. For example, in one embodiment, comparing the genre fieldof a tag 208 to a keyword database reveals that “Action” is present ineach, and that it is mapped to haptic theme information dictating thatthe overall intensity or library of haptic effects output during theplayback of a video is increased over a default level. In anotherexemplary embodiment, comparing a script 206 to a keyword databasereveals that the phrase “I love you” is present in each, and that it ismapped to haptic theme information dictating that the overall intensityof haptic effects output during the playback of a video is decreasedunder a default level. In another embodiment, comparing chapter data 212to a keyword database reveals the phrase “celebration” is present in thetitle of chapter 1 and the keyword database, and that it is mapped tohaptic theme information dictating that the overall intensity of hapticeffects output during the playback of a video is increased slightly overa default level during playback of chapter 1 of the multimedia content202. In one embodiment, identified haptic themes 416 are applied to thecombined haptic track 418 to generate final haptic track 420. In otherembodiments, one or more identified haptic themes 416 may be applied toa haptic track generated by only using one of the two techniquespreviously described. In such embodiments, one or more identified hapticthemes 416 are applied to haptic track 414 or haptic track 410 togenerate final haptic track 420. In one embodiment, the processor 110embeds the final haptic track 420 into multimedia data 200 comprisingmultimedia content 202 in any manner known to one having ordinary skillin the art. In another embodiment, the haptic track 420 is stored as afile separate from multimedia file 201 in a memory (e.g. memory 112). Insome embodiments involving streaming, an identified haptic theme 416 isapplied to portions of the haptic track as they are generated.

Identification and application of haptic themes 416 according to thepresent disclosure assists automated haptification algorithms (406and/or 408) in providing haptic effects more accurately tailored to themultimedia content. In some embodiments, however, certain keywords maybe prioritized over an identified haptic theme. For example, in oneembodiment, the intensity of a haptic effect output based on thepresence of “explosion” in the closed caption information of amultimedia file may be increased above a default intensity, therebyoverriding application of a haptic theme that is softening the intensityof haptic effects based on the presence of “romantic comedy” in thegenre field of a metadata tag embedded in the multimedia content. One ofordinary skill in the art would recognize that the words or phrases thatmay be used to determine an appropriate haptic theme are not limited tothe examples provided by the present disclosure, but rather extends toany words or phrases that may provide clues as to what events orelements are present in the multimedia content.

The first two techniques of determining haptic effects based oncomplementary information 422 discussed above may be used alone or incombination, while the technique employing haptic themes may be used inconjunction with either or both of the first two techniques, or inconjunction with any other techniques that identifies haptic effects tobe output during playback of multimedia content 422. For example, in oneembodiment complementary information 422 is generally used to providecontext to an automated haptification algorithm (406 and/or 408)employing pattern recognition techniques to the audio and/or videotracks, but for a small set of keywords present in closed caption dataparticular haptic effects are caused to be output based solely on thepresence of those keywords in the complementary information.

At step 306, haptic effects are output during playback of the multimediacontent 202. In one embodiment, a haptic track (e.g., 410, 414, 418, or420) generated in the steps above comprises data structures defining thehaptic effect to be output by haptic output device 118. In anotherembodiment, the haptic track (e.g., 410, 414, 418, or 420) containshaptic effect codes and the processor 110 uses the haptic effect codesto look up data structures defining the haptic effects in a hapticeffect database stored in memory 112. In such embodiments, processor 110transmits a signal to drive haptic output device 118 based on the hapticeffect definitions in such data structures. In another embodiment, thehaptic accessory incorporating one or more haptic output devices 118(e.g. haptic furniture, wearable haptic device) comprises a processorand a memory comprising a database mapping haptic effect identificationcodes to data structures defining haptic effects. In one suchembodiment, processor 110 transmits haptic effect codes to the processorof the haptic accessory, and then that processor uses the haptic effectcodes to look up data structures defining the haptic effects andtransmits a signal to drive haptic output device 118 based on the hapticeffect definitions in such data structures.

Commentary Audio Tracks

In some cases, movies, documentaries and other video content haveassociated commentary audio tracks that may be played simultaneouslywith the video in which directors, actors, and/or others discuss thevideo. In one embodiment of the present disclosure, commentary audiotracks are another type of complementary information 422 on which adetermination of a haptic track 420 may be based. In one suchembodiment, an associated commentary audio track has its ownsubtitle/closed caption information that is used to determine hapticeffects to be provided in a manner similar to that described above withrespect to subtitle/closed caption data 204. In another embodiment,subtitle/closed caption data for a commentary audio track is notavailable and system 100 processes the commentary audio track togenerate a textual version of the commentary audio track, in any mannerknown to those having skill in the art, that is then used to determinehaptic effects to be provided in a manner similar to that describedabove with respect to subtitle/closed caption data 204.

Advanced Methods for Providing Haptic Effects Based on ComplementaryInformation

In addition to embodiments of the present disclosure set forth abovethat generally describe matching keywords or phrases to in complementaryinformation 422 to identify predetermined haptic effects, to providecontext to automated haptification algorithms, and/or identify one ormore haptic themes to be applied, additional techniques may be used inconjunction with, or independent of, the methods described above toincrease accuracy of the generated haptic track 420. In one embodiment,the presence of a keyword/phrase in subtitle/closed caption data 204 atelement 412 is ignored if it is present in textual quotes of audiblespeech corresponding to the audio of multimedia content 202, but isrelied upon, as described in the sections above, to identify apredetermined haptic effect, to provide context to an automatedhaptification algorithm, and/or identify a haptic theme if it is presentin text describing events visually depicted by multimedia content 202.For example, in one embodiment the presence of the word “explosion” intextual quotes of subtitle/closed caption data 204 is ignored because itis unlikely for there to be an explosion at the time that a person isspeaking the word “explosion.” But if “explosion” is present in text ofsubtitle/closed caption data 204 describing scenes or events of themultimedia content 202, its presence is used to identify a predeterminedhaptic effect, to provide context to an automated haptificationalgorithm, and/or identify a haptic theme as described in the sectionsabove. Whether to ignore or rely on the presence of a keyword/phrase inany of the types of complementary information 422 described herein, maybe determined in any manner known to one of ordinary skill in the art.For example, the database(s) of keywords/phrases described above mayspecify ignoring the presence of certain keywords/phrases for certaintypes and/or sub-types of complementary information 422 (e.g. textualdescriptions and textual quotes of subtitle/closed caption data 204) andwhen to rely on the presence of certain keywords/phrases for certaintypes and/or sub-types of complementary information 422.

In another embodiment, the presence of a keyword/phrase in complementaryinformation 422 may be weighted based on the type or sub-type ofcomplementary information 422 in which it is found. In one embodiment,weighting is used by processor 110 to prioritize the insertion of onehaptic effect over another into a haptic track 414 when the processesdescribed above result in multiple haptic effects to be output at thesame time, or close in time such that the haptic effects will overlapand interfere with one another. For example, in one embodiment thepresence of “bomb” in a textual quote in subtitle/closed caption data204 is given a low weight and the presence of “cheering” in a textualdescription of a scene/event in subtitle/closed caption data 204 isgiven a high weight. In one such an embodiment, the complementaryinformation 422 comprises subtitle/closed caption data 204 containing atextual description “Crowd cheering” and a textual quote “Dan Marinothrows a bomb; touchdown Dolphins!” at contemporaneous times in thesubtitle/closed caption data 204. The processor 110 operates asdescribed above to identify predetermined haptic effects, to providecontext to automated haptification algorithms, and/or identify hapticthemes based on the presence of “bomb” and “cheering” but prioritizesthe haptic effect, context, and/or haptic theme associated with“cheering” in the textual description in the subtitle/closed captiondata 204 based on it having a higher weight. The weight assigned to thepresence of a keyword/phrase in any of the types of complementaryinformation 422 described herein may be determined in any manner knownto one of ordinary skill in the art. For example, the database(s) ofkeywords/phrases described above may specify weights for certainkeywords/phrases in certain types and/or sub-types of complementaryinformation 422 (e.g. textual descriptions and textual quotes ofsubtitle/closed caption data 204).

In another embodiment, known natural language analysis techniques areused by processor 110 to improve the correlation of the haptic track 420to the events of the multimedia content 202. For example, in oneembodiment known algorithms for natural language analysis are used tointerpret phrases in subtitle/closed caption data 204 to ascertainactual meanings with greater accuracy by taking into account syntax andsemantics of the phrase. For example, the processor 110 applies naturallanguage analysis techniques to analyze the phrase “there is no bomb” insubtitle/closed caption data 204 and, understanding the real meaning ofthe phrase, determines that the presence of “bomb” should not becompared to the database(s) of keywords and phrases to identify apredetermined haptic effect corresponding to “bomb.” However, theprocessor 110 may determine that presence of a discussion regarding a“bomb” in the subtitle/closed caption data 204 is indicative of a genreof the multimedia content, such as war or action, and select anappropriate haptic theme. In another embodiment, processor 110 appliesnatural language analysis techniques to analyze the phrase “the bombwill explode in 3, 2, 1 . . . ” in subtitle/closed caption data 204 and,recognizing the countdown in the phrase, operates to insert apredetermined haptic effect corresponding to “bomb” into a haptic track420 at a precise time or to provide context identification informationto an automatic haptification algorithm (406 and/or 408) for a precisetime based on the phrase. The foregoing are but a few examples of usingnatural language analysis techniques to analyze complementaryinformation 422. One of ordinary skill in the art would recognizenumerous applications for using natural language analysis techniques inthe context of the present disclosure.

In some instances, matching keywords or phrases in complementaryinformation 422 to identify predetermined haptic effects, to providecontext to automated haptification algorithms, and/or identify one ormore haptic themes to be applied may suffer from inaccuracy due to somewords or phrases having multiple possible meanings in a particularlanguage. In one embodiment, subtitle/closed caption information 204comprises subtitle and/or closed caption information in multiplelanguages (e.g., English, Spanish, and French) and such an ambiguity fora word/phrase in a first language (e.g., English) is resolved byanalyzing the word/phrase in one or more of the other availablelanguages (e.g., Spanish, French) where the word/phrase in one of theother languages generally has one accepted meaning. In one embodiment,this disambiguation method may be implemented by flagging ambiguouswords/phrases in each language as ambiguous in the keyword database(s)for that language. In one such embodiment, upon determining that akeyword database indicates that a word/phrase in a language isambiguous, system 100 would analyze the corresponding word/phrase in asecond language to attempt to resolve the ambiguity. For example, in oneembodiment subtitle/closed caption data 204 comprises English, French,and Spanish closed captions. In this embodiment, processor 110 comparesEnglish closed caption text to a keyword database and finds the word“tie” is present in the closed caption text and in the keyword database,but is flagged as an ambiguous word. Based on “tie” being flagged asambiguous, the processor 110 analyzes the French closed caption textaround the same time stamp in the multimedia content 202 that “tie”appeared in order to determine its meaning. In one embodiment, thisanalysis comprises looking for French synonyms of various meanings of“tie” that, once identified in the French closed captions, will indicatewhich meaning of the word “tie” is correct. For example, the analysismight identify the French word “cravate,” which means necktie, or theFrench word “match nul,” which means tie game. Once the disambiguationof “tie” is complete, the methods described above may be used toidentify a predetermined haptic effect, provide context to automatedhaptification algorithms, and/or identify one or more haptic themes tobe applied. For example, a haptic theme appropriate for sporting eventsmay be applied based on the presence of “match nul.” One of ordinaryskill in the art would recognize that this is but one of numerouspossible examples.

Remote Generation of Haptic Tracks

While the embodiments of the present disclosure set forth abovegenerally describe processor 110 of system 100 performing the analysisof the multimedia content 202 and corresponding complementaryinformation 422 to generate a haptic track (e.g., 410, 414, 418, or420), in other embodiments the analysis and generation of a haptic track(e.g., 410, 414, 418, or 420) may be performed by a another computer orserver in communication with processor 110 through a network, such asthe Internet. In one such embodiment, the remote computer or serverembeds the haptic track (e.g., 410, 414, 418, or 420) into a multimediafile 201. Processor 110 of system 100 then downloads the multimedia file201 containing the haptic track and outputs haptic effects while playingthe multimedia content 202 as described above in relation to step 306 ofFIG. 3. In another embodiment, the remote computer stores the haptictrack (e.g., 410, 414, 418, or 420) as a file separate from multimediafile 201 locally in a memory of the remote computer. In anotherembodiment, the remote computer transmits the haptic track (e.g., 410,414, 418, or 420) to a second remote computer where it is stored as afile separate from multimedia file 201. In such embodiments, processor110 of system 100 downloads the haptic track (e.g., 410, 414, 418, or420) from the remote computer on which it is stored and outputs hapticeffects while playing the multimedia content 202 corresponding to thehaptic track as described above in relation to step 306 of FIG. 3.

Pre-Generation of Haptic Tracks

While the embodiments of the present disclosure described abovegenerally describe generating haptic tracks when a user plays multimediacontent, in other embodiments haptic tracks are generated according tothis disclosure when multimedia content is downloaded to system 100, butbefore playback. In another embodiment, haptic tracks are generatedaccording to this disclosure by a creator or distributor of multimediacontent prior to making the content available for use.

Providing Haptic Effects Based on Viewer Information

While the embodiments of the present disclosure above describe providinghaptic effects based on complementary information 422 for a multimediacontent 202, in other embodiments the generated haptic track 420 isfurther based on viewer position, posture, reaction, and biometrics(represented by additional information 424) analyzed at element 412.Viewer information may be determined using one or more of the sensorsdescribed above with respect to system 100.

In one embodiment, a haptic theme modulating the output level of hapticeffects to be above or lower than a default level is determined based ona viewer's proximity to an output source for the playback of themultimedia content 202 (e.g. A/V receiver 122, display 116 and/orspeakers 120 of system 100). For example, in one embodiment, if aviewer's proximity is approximately 10 feet from the output source forthe playback of the multimedia content, no haptic theme is selected.However, if the viewer's proximity to the output source is approximately9 feet, a haptic theme specifying an output level for the haptic effectsabove the default level is selected. As the proximity of the viewer tothe output source grows smaller, the greater the modulation of theoutput level of the haptic effects that is specified by the haptic themeselected. On the other hand, if the viewer's proximity to the outputsource is approximately 9 feet, a haptic theme specifying an outputlevel for the haptic effects below the default level is selected. And asthe proximity of the viewer to the output source increase, the lower themodulation of the output level of the haptic effects that is specifiedby the haptic theme selected.

In a similar embodiment, a haptic theme modulating the output level ofhaptic effects to be above or lower than a default level is determinedbased on a viewer's viewing angle to an output source for the playbackof the multimedia content 202 (e.g. display 116 of system 100). Forexample, in one embodiment, if a viewer's viewing angle is approximately90° with respect to the output source for the playback of the multimediacontent (directly in front), haptic effects are output at their maximumlevel according to the haptic track 420. However, as a viewer's viewingangle to the output source moves closer to 0°, the haptic effectsspecified by a haptic track 420 are modulated to a lower output level.

In another embodiment, a viewer's position, posture and or movement isused to determine a haptic theme. For example, in one embodiment ahaptic theme for horror or action movie is selected based on a viewercovering his/her eyes. In another embodiment, a haptic theme for aromantic comedy is selected based on a couple sitting close to eachother in a relaxed manner and smiling.

In still another embodiment, viewer biometric information is used toselect a theme. For example, in one embodiment a haptic themecorresponding to an action movie is selected based on a viewer having anelevated heart rate indicating stress and/or excitement. In anotherembodiment, a haptic theme modulating the haptic effects to a lowerlevel is selected where viewer heart rates indicate the viewer isrelaxed.

Video Games

In one embodiment, multimedia content 202 comprises audio and/or videooutput during playing of a video game. For example, in one embodiment avideo game comprises multimedia content 202 in the form of a videosequence and complementary information 422 comprising any of the typesdiscussed above in relation to multimedia content comprising video (e.g.subtitle/closed caption data 204 corresponding to the video sequence).Accordingly, the embodiments for providing haptic effects based oncomplementary information 422 discussed above also apply to video games.However, in further embodiments other information (represented byadditional information 424) may be used independently, or in addition tocomplementary information 422 discussed in the embodiments above, toidentify haptic effects to be output.

In one embodiment, system 100 is configured to execute video gameprograms and provides an application program interface that video gameprograms may call to initiate playback of an audio file (“Audio API”).In one such embodiment, a textual identifier (e.g., a file name) of theaudio file requested for playback through the Audio API (“file namedata”) is provided to the Audio API. In one embodiment, system 100 atelement 412 may analyze the filename data to 1) identify haptic effectsto be output based on keywords; 2) provide contextual information toalgorithms/processes for automatically haptifying audio and/or videoimage; and 3) identify a haptic theme in a similar manner to thatdescribed above with respect to various types of complementaryinformation 422. For example, in one embodiment the file name data isanalyzed by comparing words in the file name data to a keyword databaseat element 412. In one such embodiment, the file name data comprises thetext “thunder.mp3” and comparing the “thunder” to a keyword databasereveals that “thunder” is present therein and that it is mapped tohaptic effect identification information for an intense pulsatingvibration haptic (emulating thunder). In another exemplary embodiment,the file name data comprises the text “scream.mp3” and comparing“scream” to a keyword database reveals that “scream” is present thereinand that it is mapped to context identification information used toinstruct an automated haptification algorithm (406 and/or 408) toprioritize recognition of screams, shrieks, and other sounds commonlyassociated with a scary scene. In another embodiment, comparing “scream”(from “scream.mp3”) to a keyword database reveals that “scream” ispresent in each, and that it is mapped to haptic theme informationdictating that the overall intensity or library of haptic effects outputduring the playback of a video is increased over a default level asappropriate for a scary video sequences.

In another embodiment, system 100 provides an application programinterface that video game programs may call to cause text displayed onscreen during a video game to be output as audio (“Text-to-Speech API”).In one such embodiment, the text displayed on the screen (e.g.,instructions, signs or other text displayed in the video game scene,comic-style action indicators such as “BAM” and “THWACK”) is provided tothe Text-to-Speech API. In one embodiment, system 100 may analyze atelement 412 the displayed to 1) identify haptic effects to be outputbased on keywords; 2) provide contextual information toalgorithms/processes for automatically haptifying audio and/or videoimage; and 3) identify a haptic theme based on in a similar manner tothat described above with respect to various types of complementaryinformation 422 (e.g., subtitle/closed caption data 204).

General

The foregoing description of some embodiments of the invention has beenpresented only for the purpose of illustration and description and isnot intended to be exhaustive or to limit the invention to the preciseforms disclosed. Numerous modifications and adaptations thereof will beapparent to those skilled in the art without departing from the spiritand scope of the invention.

Reference herein to “one embodiment” or “an embodiment” means that aparticular feature, structure, operation, or other characteristicdescribed in connection with the embodiment may be included in at leastone implementation of the invention. The invention is not restricted tothe particular embodiments described as such. The appearance of thephrase “in one embodiment” or “in an embodiment” in various places inthe specification does not necessarily refer to the same embodiment. Anyparticular feature, structure, operation, or other characteristicdescribed in this specification in relation to “one embodiment” may becombined with other features, structures, operations, or othercharacteristics described in respect of any other embodiment.

We claim:
 1. A method comprising: receiving, by a processor, a datastream comprising a video track and complementary information;determining, by the processor, a haptic effect based on the video track;determining, by the processor, a modified version of the haptic effectbased on the complementary information; and transmitting, by theprocessor, a signal configured to cause a haptic output device to outputthe modified version of the haptic effect while playing the video track.2. The method of claim 1, further comprising using the complementaryinformation to provide context to an automated haptification algorithm.3. The method of claim 1, wherein the complementary informationcomprises a haptic effect code corresponding to one or more hapticeffects in a database.
 4. The method of claim 1, further comprising:synchronizing output of the modified version of the haptic effect withan event in the video track; and outputting the modified version of thehaptic effect concurrently with the event in the video track.
 5. Themethod of claim 1, wherein the complementary information comprises atleast one of a title, artist, author, genre, rating, review, list ofsongs, comment, chapter data, or song type.
 6. The method of claim 1,further comprising: determining a haptic theme using the complementaryinformation; and generating the modified version of the haptic effect bymodifying a characteristic of the haptic effect based on the haptictheme.
 7. The method of claim 1, wherein the data stream furthercomprises an audio track corresponding to the video track and thecomplementary information.
 8. A system comprising: a processing device;and a memory device on which instructions executable by the processingdevice are stored to cause the processing device to: receive a datastream comprising a video track and complementary information; determinea haptic effect based on the video track; determine a modified versionof the haptic effect based on the complementary information; andtransmit a signal configured to cause a haptic output device to outputthe modified version of the haptic effect while playing the video track.9. The system of claim 8, wherein the memory device further includesinstructions executable by the processing device to cause the processingdevice to: use the complementary information to provide context to anautomated haptification algorithm.
 10. The system of claim 8, whereinthe complementary information comprises a haptic effect codecorresponding to one or more haptic effects in a database.
 11. Thesystem of claim 8, wherein the memory device further includesinstructions executable by the processing device to cause the processingdevice to: synchronize output of the modified version of the hapticeffect with an event in the video track; and cause the modified versionof the haptic effect to be output concurrently with the event in thevideo track.
 12. The system of claim 8, wherein the complementaryinformation comprises at least one of a title, artist, author, genre,rating, review, list of songs, comment, chapter data, or song type. 13.The system of claim 8, wherein the memory device further includesinstructions executable by the processing device to cause the processingdevice to: determine a haptic theme using the complementary information;and generate the modified version of the haptic effect by modifying acharacteristic of the haptic effect based on the haptic theme.
 14. Thesystem of claim 8, wherein the data stream further comprises an audiotrack corresponding to the video track and the complementaryinformation.
 15. A non-transitory computer readable medium comprisingprocessor-executable program code, which when executed by a processor isconfigured to cause the processor to: receive a data stream comprising avideo track and complementary information; determine a haptic effectbased on the video track; determine a modified version of the hapticeffect based on the complementary information; and transmit a signalconfigured to cause a haptic output device to output the modifiedversion of the haptic effect while playing the video track.
 16. Thenon-transitory computer readable medium of claim 15, further comprisingprocessor-executable program code that is executable by the processor tocause the processor to: use the complementary information to providecontext to an automated haptification algorithm.
 17. The non-transitorycomputer readable medium of claim 15, wherein the complementaryinformation comprises a haptic effect code corresponding to one or morehaptic effects in a database.
 18. The non-transitory computer readablemedium of claim 15, further comprising processor-executable program codethat is executable by the processor to cause the processor to:synchronize output of the modified version of the haptic effect with anevent in the video track; and cause the modified version of the hapticeffect to be output concurrently with the event in the video track. 19.The non-transitory computer readable medium of claim 15, wherein thecomplementary information comprises at least one of a title, artist,author, genre, rating, review, list of songs, comment, chapter data, orsong type.
 20. The non-transitory computer readable medium of claim 15,further comprising processor-executable program code that is executableby the processor to cause the processor to: determine a haptic themeusing the complementary information; and generate the modified versionof the haptic effect by modifying a characteristic of the haptic effectbased on the haptic theme.